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Abstract 

This study describes and illustrates a modification of the Delphi technique that was designed to 
extend its applicability to large competency data sets. The approach consisted of rotating sub- 
sets of a larger set of competencies through three sub -panels and then examining the viability of 
the approach. The context of the study was the identification of competencies needed to update 
training requirements for 21st century plastering contractors. Based on the outcomes of this 
study, the rotational approach represents a promising mechanism for extending the usefulness of 
the Delphi tool to larger data sets. The findings also indicate that the factors that erode the 
validity and usefulness of traditional modified Delphi studies also negatively affect the rotational 
approach. 

An obstacle facing many educators has been identifying an appropriate research methodology 
for developing curriculum when teaching vocational subjects. This is especially true when a 
research topic is very broad and when new curricular areas are being developed. In such 




situations there is often a large number of competencies to be evaluated by recognized experts in 
the content area. A number of techniques have been used that are generally associated with the 
competency-based approach to curriculum development. These include such tools as 
occupational analysis, DACUM (Developing a Curriculum), and the Delphi technique (Finch & 
Crunkilton, 1989; Frykland, 1970; Miller, 1990; Rothwell & Kazanas, 1992). These techniques 
span a range of approaches from observing workers on the job to conducting meetings with 
workers to conducting analyses based on work previously conducted on related occupations 
(Blank, 1982). Each technique has advantages and disadvantages, especially in dealing with 
diverse groups of experts involving relatively large data sets. 

This manuscript describes a modification developed for use with the Delphi technique, which 
was selected for use in a curriculum competency study. The Delphi technique was selected for 
use in this study due to its ability to obtain expert input from individuals who were widely 
dispersed geographically. This technique has been used in a number of fields for long-range 
planning- including education, international affairs, transportation, leisure activities and the like 
(McCampbell & Stewart, 1992). 

The Delphi survey technique was developed in the 1950s by two researchscientists working at 
The Rand Corporation, Olaf Helmer and Norman Dalkey. They developed the procedure as a 
tool for forecasting future events using a series of intensive questionnaires interspersed with 
controlled- opinion feedback (Dalkey & Helmer, 1993; McCampbell & Stewart, 1992; Weaver, 
1971). Participants were solicited experts in the issues related to national defense such as 
forecasting probable bombing targets the Russian government might choose in the event of an 
attack on the United States (Dalkey & Helmer, 1963). 

The Delphi begins with an open-ended questionnaire that is given to a panel of selected experts 
to solicit specific information about a subject or content area. In subsequent rounds of the 
procedure, participants rate the relative importance of individual items and also make changes to 
the phrasing or substance of the items. Through a series of rounds (typically three) the process is 
designed to yield consensus. 

The modified Delphi technique is similar to the full Delphi in terms of procedure (i.e., a series of 
rounds with selected experts) and intent (i.e., to predict future events and to arrive at consensus). 
The major modification consists of beginning the process with a set of carefully selected items. 
These pre-selected items may be drawn from various sources including related competency 
profiles, synthesized reviews of the literature, and interviews with selected content experts. The 
primary advantages of this modification to the Delphi is that it (a) typically improves the initial 
round response rate, and (b) provides a solid grounding in previously developed work. 

Additional advantages related to the use of the modified Delphi technique include reducing the 
effects of bias due to group interaction, assuring anonymity, and providing controlled feedback 
to participants (Dalkey 1972a, 1972b, & Judd 1972). Brooks (1979) noted that three mailings are 
usually sufficient in order to arrive at consensus. 

Additional Modifications to the Delphi Technique 

One significant problem that is often encountered in Delphi studies has to do with the rigors 
involved in maintaining focus when rating competency sets containing large numbers of items. 
Beyond problems with maintaining sufficient levels of concentration, large competency sets can 
consume large blocks of time and thus represent a common source of panel attrition. Many 
Delphi studies contain as many as 50 items; however, some studies contain considerably more 
than that amount. 

This article reports on a research study that employed an additional modification to the modified 


Delphi technique. This was termed a "Rotational" Delphi technique and was specifically 
designed to explore the viability of a procedure for rotating subsets of larger competency sets 
through sub -panels in order to reduce the level of fatigue on panelists and to increase the volume 
of competencies that can be effectively and efficiently studied using the Delphi procedure. 

Context and Purpose of the Study 

The research study that used the rotational technique was conducted to identify competencies 
needed to update training requirements for 21st century plastering contractors (i.e., lath, plaster, 
dry wall, metal- stud framing, fire proofing insulation, exterior insulation finish systems, 
specialty, and decorative & ornamental plastering). Due to changes in the construction industry 
that have occurred over the last two decades as a result of new technology and management 
practices, a competency analysis was needed to develop a foundation for curriculum and training 
program development for use in union and nonunion sectors (Scarcella, 1997). 

The purpose of this article is to describe and illustrate the viability of using the rotational Delphi 

procedure through an actual study of competencies needed for 21 st century plastering 
contractors. The specific question that provided focus for the study was, Does the rotational 
modification to the Delphi procedure represent a valid and efficient means of identifying and 
validating large competency sets in occupationally- specific areas? 


Procedures and Methodology 
Sample 

A sample of 22 contractors and 22 policy makers from selected contractor associations and 
organizations was asked to verify and rate the relative importance of the set of identified 
competency items. The research goal was to obtain consensus regarding what competencies are 
important for plastering construction contractors. A descriptive research design was utilized to 
achieve this objective because the panel members under investigation were attempting to 
describe what knowledge, skills, and abilities are associated with plastering contractors (Crowl, 
1993). 

It was determined that contractors and policy makers from different regions of the country, as 
well as industry experience and perspective would provide some breadth of insight into 
certification requirements for plastering contractors. The rationale for using the two different 
types of panelists to react to the common competencies list was that, although both groups come 
from different parts of the industry, they represent unique perspectives of expert opinion. Both 
perspectives are important as certification, licensing, and curriculum programs are developed to 
educate the workforce. 

Three rounds of the modified Rotational Delphi technique were conducted. Each round 
presented an instrument to panel members who completed and returned it to the researcher. The 
responses were analyzed and compiled to build the next round's instrument. For each item, 
interquartile ranges were calculated as measures of dispersion and median scores were calculated 
as measures of central tendency. The combination of these indices was used to determine the 
degree of importance and consensus for each item. 

Instrumentation 


The Plastering Contractors Competency Rating Scale (PCCRS) was developed for use in the 
study. The procedures for developing the instrument began with identifying and collecting 
documents pertinent to the certification requirements for plastering contractors. This included 


sources such as licensure examinations, textbooks, apprenticeship training materials, and job 
descriptions. Based on a table of specifications developed from the review of the literature, 75 
competencies were selected and divided into two major classifications: 48 having to do 
managerial functions (e.g., accounting, hiring, personnel, training, etc.) and 27 concentrating in 
technical areas (e.g., skills associated with the application of materials, use of tools, etc.). The 
instrument was subjected to scrutiny by a panel of experts and was pilot tested by a sample of 
plastering contractors and policy makers. Items were rated on a scale of 1 important to 6=Not 
Important. 

After the instrument had been validated and pilot tested, it was subdivided into three forms. Each 
form of the instrument contained two-thirds of the entire competency set. This was done by 
systematically eliminating every third item as illustrated in Table 1. This procedure provided a 
mechanism for exposing panelists to every item in the total competency set at least once during 
the first two rounds of the modified Delphi process. An alternative method of developing sub- 
forms of the instrument would have been to have divided the competency set into three forms 
with each form containing unique items (i.e., one-third of the total set with no duplication of 
items across forms). The rationale for using the first approach (i.e., two-thirds of the total set on 
each form of the instrument) was to reduce the total number of items rated during each round 
while maintaining some continuity of ratings across the entire panel. 

Table 1 

Instrument Master Data Rotation For Rounds One and Two 


Master Form 




Item Number 

Form One 

Form Two 

Form Three 

Item 1 




Item 2 

X 


X 

Item 3 

X 

X 


Item 4 


X 

X 

Item 5 

X 


X 

Item 6 

X 

X 


Item 7 


X 

X 


X 


X 


The Rotational Process 


The total panel was randomly subdivided into three sub-panels on a stratified basis (e.g., equal 
numbers of contractors and policy makers in each of the three groups). The three forms of the 
instrument were then "rotated" through the subpanels on a systematic basis during rounds one 


and two (see Figure 1). The round three instrument was comprised of all items that had not 
reached consensus by the end of the second round. This involved 13 (27%) of the managerial 
items and 19 (70%) of the technical items (i.e., 43% of the total competency set). 


Round One Round Two Round Three 

FI F2 F3 F3 FI F2 C 


G1 G2 G3 G1 G2 G3 CP 


Figure 1 . Illustration of the rotational process. 

Note: F = 3 forms of 1 instrument 
G = Groups of panelists 

C = Composite forms of instrument comprised of all non- consensus items 
CP = Combined panel 

Data Gathering Procedures 

During the first two rounds, each of the three sub-panels received a form of the instrument 
containing 50 items along with a cover letter that included an explanation of the study and the 
six point rating scale. In addition to rating each of the items, raters were asked to make 
additional suggestions related to (a) the phrasing of items or (b) ideas about any new items that 
they believed should be added. The only difference between the first and second round 
procedure was that interquartile ranges of the first round ratings were placed on each of the 
second round items. 

If any of the panelists second round ratings fell outside of the interquartile range (indicated in 
brackets), they were asked to provide a brief explanation of their rating in the comments section 
that was located on the back side of each page. If panel members agreed with the majority 
rating, no additional commentary was required. Panelists were also requested to make written 
comments or reword those items, which lacked clarity as part of the feedback on the second 
round instruments (if applicable). In round three, interquartile ranges were placed in brackets 
directly on the instrument along with any comments obtained during round two. 

Data Analysis Procedures 

The data analysis process included three primary components. The first component (and primary 
consideration of the Delphi process) consisted of analyzing each item for consensus. For this 
study, consensus was considered to have been achieved when an Interquartile Range score of 
less than 1.2 was obtained (Zeliff & Heldenbrand, 1993). A second component of the analysis 
was to evaluate the perceived importance of the items. To accomplish this, the six-point scale 
was evenly divided into high, medium, and low importance. Items were then classified into one 
of six categories based on an analysis of consensus combined with importance. 

The third component of the analysis process consisted of examining the effectiveness of the 


rotational modification to the Delphi process. This process consisted of comparing the 
interquartile range scores of first and second time raters on each item at the end of round two. 
As noted previously, during round two, one-half of the items that each group received were 
being rated by that group for the second time. The other half of the items was new to each group 
(rotated onto the group from the first round). 

The analysis process was designed to compare the second round ratings of "first" and "second" 
time raters. The rationale for this procedure was that strong similarities between the two groups 
could suggest that little consensus- building information was lost when all competencies were not 
rated by all panelists during the initial round. Strong dissimilarities, on the other hand, would 
suggest that the rotational process had not worked effectively. 

Findings 

The analysis procedure consisted of examining the differences in interquartile ratings for each 
item. Separate frequency and percentage scores were compiled for managerial skills (see Table 
2) and technical skills (see Table 3). For managerial skills, there was no difference between first 
and second time raters on 50% of the items. An interquartile range difference greater than one 
was detected on only one item. It is important to note that there was strong consensus across the 
panel on the managerial skill set. By the end of round three, raters for 95.8% of the managerial 
skill set had achieved consensus. Therefore, it is likely that the low interquartile rating 
differences between first and second time raters was to some extent a function of the strong 
overall consensus on the managerial item set. 

Table 2 

Difference in Managerial Skills Competency Items/Interquartile Ratings Between First and 
Second Time Raters 


Interquartile Differences 

Frequency 

% 

.0 

23 

48.9 

.25 

5 

11.6 

.50 

0 

0.0 

.75 

3 

6.6 

1.00 

15 

31.9 

1.25 

0 

0.0 

1.50 

0 

0.0 

1.75 

0 

0.0 

2.00 

1 

2.8 

N = 

47 

100.0 


I 


Note: 36 out of 46 (78 .2% 1 first time rater's managerial skills interquartile ranges were equal or 
lower than second time raters. 

24 out of 46 (52.1%) second time rater’s managerial skills interquartile ranges were equal or 
higher than first time raters. 

The differences between the interquartile ratings of the technical skill set items were somewhat 
greater. Seventy-two percent of the items were within an interquartile range of one while the 
difference between first and second time raters was greater than one for seven items in the set 
(see Table 3). The degree of consensus achieved by raters for the technical skill set at the end of 
round three was less (44.4%) than with the managerial skills. Thus, it could be expected that this 
would translate into greater interquartile differences between first and second raters than was the 
case with the managerial skill comparisons. 

Table 3 

Difference in Technical Skills Competency Items/Interquartile Ratings Between First and 
Second Time Raters 


Interquartile Differences 

Frequencv 

% 

.0 

4 

16.0 

.25 

4 

16.0 

.50 

1 

4.0 

.75 

2 

8.0 

1.00 

7 

28.0 

1.25 

1 

4.0 

1.50 

1 

4.0 

1.75 

0 

0.0 

2.00 

5 

20.0 

N = 

25 

100.00 


Note: 15 out of 25 (60.0%t first time rater's technical skills interquartile ranges were equal or 
lower than second time raters. 

14 out of 25 (56.0%) second time rater's technical skills interquartile ranges were equal or higher 
than first time raters. 

The relationship between the extent of total group consensus and first/second round rater 


comparisons requires additional comment, particularly as it relates to interpreting the results 
from the technical skills component of this analysis. Whereas high consensus corresponded 
closely with high first and second round comparisons with the managerial skills, a relatively low 
level of consensus yielded a relatively high degree of first and second round correspondence for 
the technical skills. This pattern of relationships for the technical skills makes sense, given that 
(a) high interquartile range scores indicates a lack of consensus, and (b) a relatively high 
percentage of raters agreed with the relative lack of consensus. The correspondence scores (72% 
within one interquartile) between first and second round raters is relatively high given the 
relatively low degree of consensus. In other words, the first and second time raters tended to 
agree on a relative lack of consensus. 

Discussion 

This study was designed to explore the feasibility of using a rotational modification to the Delphi 
process in order to expand the total number of competencies that can be analyzed effectively. 
The analysis of the rotational modification was conducted as part of a study of competencies 
needed to update training requirements for 21st century plastering contractors. Based on the 
findings of the study and an analysis of the rotational process, the following comments and 
observations can be reported. 

1 . The validity of the modified Delphi process depends on the careful and systematic 
application of procedures for initial competency selection (e.g., reviewing the literature, 
developing a table of specifications, conducting a pilot test, etc.). In a traditional modified 
Delphi, this careful selection process is necessary in order to (a) avoid biasing panelists by 
including inappropriate or unnecessary items and (b) increase the probability that 
consensus can be achieved in an efficient and timely manner. For the rotational 
modification, the careful initial selection is important because it increases the probability 
that first and second time raters would rate individual items in somewhat the same way. 

2. The extent to which the Delphi process is capable of achieving consensus is a function, 
not only of the quality of the initial competency selection process, but also of the degree 
of controversy or clarity that exists in a given content area or profession. In this study of 
the plastering industry, the Delphi provided a mechanism for refining the understanding of 
the skills required to accomplish the job. Given this, the use of a rotational modification 
for emerging professions or controversial content areas is not recommended. 

3. As a corollary to the previous point, in those situations where the Delphi is being used as 
a refinement tool and where the required pool of experts is readily available (as was the 
case in this study), the rotational modification represents a promising approach for 
expediting the consensus process while reducing the workload on panelists. 

4. There are limitations to the rotational delphi procedure which should be acknowledged. 
Specifically, since all round two instruments contained brackets indicating interquartile 
ratings for round one, first time raters (during round two) could well have been biased by 
the previous first round ratings. In this study, where the items were relatively 
homogeneous in terms of consensus (particularly on the management set), the concern 
with bias is less problematic. 

5. On reflection, it seems clear that the same factors that threaten the validity of a pure or 
modified Delphi (non-rotational) are nearly identical to those that erode the quality of a 
rotational Delphi; namely, (a) a lack of expertise on the panel, (b) lack of clear content 
definition, and (c) a poorly developed initial data set (for the modified Delphi). Stated in 
the reverse, we may reasonably conclude that Delphi studies that are conducted according 
to established criteria and procedures are candidates for the rotational modification. 


In conclusion, the rotational modification to the Delphi process represents a viable alternative to 

the established modified Delphi as a mechanism for use with large data sets. The rotational 

procedure that was used in this study should be viewed as a preliminary and exploratory process 

that holds promise for refinement and use in other Delphi studies. 
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